Active Learning to Maximize Area Under the ROC Curve

In active learning, a machine learning algorithmis given an unlabeled set of examples U, and is allowed to request labels for a relatively small subset of U to use for training. The goal is then to judiciously choose which examples in U to have labeled in order to optimize some performance criterion, e.g. classification accuracy. We study how active learning affects AUC. We examine two existing algorithms from the literature and present our own active learning algorithms designed to maximize the AUC of the hypothesis. One of our algorithms was consistently the top performer, and Closest Sampling from the literature often came in second behind it. When good posterior probability estimates were available, our heuristics were by far the best.

SAS macros for point and interval estimation of area under the receiver operating characteristic curve for non-proportional and proportional hazards Weibull models

Aims and objectives  For prediction of risk of cardiovascular end points using survival models the proportional hazards assumption is often not met. Thus, non-proportional hazards models are more appropriate for developing risk prediction equations in such situations. However, computer program for evaluating the prediction performance of such models has been rarely addressed. We therefore developed SAS macro programs for evaluating the discriminative ability of a non-proportional hazards Weibull model developed by Anderson (1991) and that of a proportional hazards Weibull model using the area under receiver operating characteristic (ROC) curve.

Method  Two SAS macro programs for non-proportional hazards Weibull model using Proc NLIN and Proc NLP respectively and model validation using area under ROC curve (with its confidence limits) were written with SAS IML language. A similar SAS macro for proportional hazards Weibull model was also written.

Results  The computer program was applied to data on coronary heart disease incidence for a Framingham population cohort. The five risk factors considered were current smoking, age, blood pressure, cholesterol and obesity. The predictive ability of the non-proportional hazard Weibull model was slightly higher than that of its proportional hazard counterpart. An advantage of SAS Proc NLP in terms of the example provided here is that it provides significance level for the parameter estimates whereas Proc NLIN does not.

Conclusion  The program is very useful for evaluating the predictive performance of non-proportional and proportional hazards Weibull models.

Model Checking for ROC Regression Analysis

The Receiver Operating Characteristic (ROC) curve is a prominent tool for characterizing the accuracy of continuous diagnostic test. To account for factors that might invluence the test accuracy, various ROC regression methods have been proposed. However, as in any regression analysis, when the assumed models do not fit the data well, these methods may render invalid and misleading results. To date practical model checking techniques suitable for validating existing ROC regression models are not yet available. In this paper, we develop cumulative residual based procedures to graphically and numerically assess the goodness-of-fit for some commonly used ROC regression models, and show how specific components of these models can be examined within this framework. We derive asymptotic null distributions for the residual process and discuss resampling procedures to approximate these distributions in practice. We illustrate our methods with a dataset from the Cystic Fibrosis registry.

ROC curves of obesity indicators have a predictive value for children hypertension aged 7-17 years

Objective: The aim of the study is to examine the distribution of integrated covariate and its association with blood pressure (BP) among children in Anhui province, China, and assess the predictive value of integrated covariate to children hypertension. Methods: A total of 2,828 subjects (1,588 male and 1,240 female) aged 7-17 years participated in this study. Height, weight, waistline, hipline and BP of all subjects were measured, obesity and overweight were defined by an international standard, specifying the measurement, the reference population, and the age and sex specific cut off points. High BP status was defined as systolic blood pressure (SBP) and/or diastolic blood pressure (DBP) > 95th percentile for age and gender. Results: Our results revealed that the prevalence of children hypertension was 11.03%, the SBP and DBP of obesity group were significantly higher than that of normal group. Anthropometric obesity indices such as body mass index (BMI) were positively correlated with SBP and DBP. Integrated covariate had a better performance than the single covariate in the receiver-operating characteristic (ROC) curve, the cut-off value; the sensitivity and the specificity of the integrated covariate were 0.112, 0.577, 0.683, respectively. Conclusion: Integrated covariate is a simple and effective anthropometric index to identify childhood hypertension.

Development and validation of 3-Minute Nutrition Screening (3-MinNS) tool for acute hospital patients in Singapore

It is important to detect and treat malnutrition in hospital patients so as to improve clinical outcome and reduce hospital stay. The aim of this study was to develop and validate a nutrition screening tool with a simple and quick scoring system for acute hospital patients in Singapore. In this study, 818 newly admitted patients aged above 18 years old were screened using five parameters that contribute to the risk of malnutrition. A dietitian blinded to the nutrition screening score assessed the same patients using the reference standard, Subjective Global Assessment (SGA) within 48 hours. The sensitivity and specificity were established using the Receiver Operator Characteristics (ROC) curve and the best cutoff scores determined. The nutrition parameter with the largest Area Under the ROC Curve (AUC) was chosen as the final screening tool, which was named 3-Minute Nutrition Screening (3-MinNS). The combination of the parameters weight loss, intake and muscle wastage (3-MinNS), gave the largest AUC when compared with SGA. Using 3-MinNS, the best cutoff point to identify malnourished patients is three (sensitivity 86%, specificity 83%). The cutoff score to identify subjects at risk of severe malnutrition is five (sensitivity 93%, specificity 86%). 3-Minute Nutrition Screening is a valid, simple and rapid tool to identify patients at risk of malnutrition in Singapore acute hospital patients. It is able to differentiate patients at risk of moderate malnutrition and severe malnutrition for prioritization and management purposes.

Exploring visual features through Gabor representations for facial expression detection

Gabor representations have been widely used in facial analysis (face recognition, face detection and facial expression detection) due to their biological relevance and computational properties. Two popular Gabor representations used in literature are: 1) Log-Gabor and 2) Gabor energy filters. Even though these representations are somewhat similar, they also have distinct differences as the Log-Gabor filters mimic the simple cells in the visual cortex while the Gabor energy filters emulate the complex cells, which causes subtle differences in the responses. In this paper, we analyze the difference between these two Gabor representations and quantify these differences on the task of facial action unit (AU) detection. In our experiments conducted on the Cohn-Kanade dataset, we report an average area underneath the ROC curve (A`) of 92.60% across 17 AUs for the Gabor energy filters, while the Log-Gabor representation achieved an average A` of 96.11%. This result suggests that small spatial differences that the Log-Gabor filters pick up on are more useful for AU detection than the differences in contours and edges that the Gabor energy filters extract.

Application of data mining techniques in the prediction of coronary artery disease : use of anaesthesia time-series and patient risk factor data

The high morbidity and mortality associated with atherosclerotic coronary vascular disease (CVD) and its complications are being lessened by the increased knowledge of risk factors, effective preventative measures and proven therapeutic interventions. However, significant CVD morbidity remains and sudden cardiac death continues to be a presenting feature for some subsequently diagnosed with CVD. Coronary vascular disease is also the leading cause of anaesthesia related complications. Stress electrocardiography/exercise testing is predictive of 10 year risk of CVD events and the cardiovascular variables used to score this test are monitored peri-operatively. Similar physiological time-series datasets are being subjected to data mining methods for the prediction of medical diagnoses and outcomes. This study aims to find predictors of CVD using anaesthesia time-series data and patient risk factor data. Several pre-processing and predictive data mining methods are applied to this data. Physiological time-series data related to anaesthetic procedures are subjected to pre-processing methods for removal of outliers, calculation of moving averages as well as data summarisation and data abstraction methods. Feature selection methods of both wrapper and filter types are applied to derived physiological time-series variable sets alone and to the same variables combined with risk factor variables. The ability of these methods to identify subsets of highly correlated but non-redundant variables is assessed. The major dataset is derived from the entire anaesthesia population and subsets of this population are considered to be at increased anaesthesia risk based on their need for more intensive monitoring (invasive haemodynamic monitoring and additional ECG leads). Because of the unbalanced class distribution in the data, majority class under-sampling and Kappa statistic together with misclassification rate and area under the ROC curve (AUC) are used for evaluation of models generated using different prediction algorithms. The performance based on models derived from feature reduced datasets reveal the filter method, Cfs subset evaluation, to be most consistently effective although Consistency derived subsets tended to slightly increased accuracy but markedly increased complexity. The use of misclassification rate (MR) for model performance evaluation is influenced by class distribution. This could be eliminated by consideration of the AUC or Kappa statistic as well by evaluation of subsets with under-sampled majority class. The noise and outlier removal pre-processing methods produced models with MR ranging from 10.69 to 12.62 with the lowest value being for data from which both outliers and noise were removed (MR 10.69). For the raw time-series dataset, MR is 12.34. Feature selection results in reduction in MR to 9.8 to 10.16 with time segmented summary data (dataset F) MR being 9.8 and raw time-series summary data (dataset A) being 9.92. However, for all time-series only based datasets, the complexity is high. For most pre-processing methods, Cfs could identify a subset of correlated and non-redundant variables from the time-series alone datasets but models derived from these subsets are of one leaf only. MR values are consistent with class distribution in the subset folds evaluated in the n-cross validation method. For models based on Cfs selected time-series derived and risk factor (RF) variables, the MR ranges from 8.83 to 10.36 with dataset RF_A (raw time-series data and RF) being 8.85 and dataset RF_F (time segmented time-series variables and RF) being 9.09. The models based on counts of outliers and counts of data points outside normal range (Dataset RF_E) and derived variables based on time series transformed using Symbolic Aggregate Approximation (SAX) with associated time-series pattern cluster membership (Dataset RF_ G) perform the least well with MR of 10.25 and 10.36 respectively. For coronary vascular disease prediction, nearest neighbour (NNge) and the support vector machine based method, SMO, have the highest MR of 10.1 and 10.28 while logistic regression (LR) and the decision tree (DT) method, J48, have MR of 8.85 and 9.0 respectively. DT rules are most comprehensible and clinically relevant. The predictive accuracy increase achieved by addition of risk factor variables to time-series variable based models is significant. The addition of time-series derived variables to models based on risk factor variables alone is associated with a trend to improved performance. Data mining of feature reduced, anaesthesia time-series variables together with risk factor variables can produce compact and moderately accurate models able to predict coronary vascular disease. Decision tree analysis of time-series data combined with risk factor variables yields rules which are more accurate than models based on time-series data alone. The limited additional value provided by electrocardiographic variables when compared to use of risk factors alone is similar to recent suggestions that exercise electrocardiography (exECG) under standardised conditions has limited additional diagnostic value over risk factor analysis and symptom pattern. The effect of the pre-processing used in this study had limited effect when time-series variables and risk factor variables are used as model input. In the absence of risk factor input, the use of time-series variables after outlier removal and time series variables based on physiological variable values’ being outside the accepted normal range is associated with some improvement in model performance.

The crash involvement of older drivers is associated with their hazard perception latencies

Hazard perception in driving is the one of the few driving-specific skills associated with crash involvement. However, this relationship has only been examined in studies where the majority of individuals were younger than 65. We present the first data revealing an association between hazard perception and self-reported crash involvement in drivers aged 65 and over. In a sample of 271 drivers, we found that individuals whose mean response time to traffic hazards was slower than 6.68 seconds (the ROC-curve derived pass mark for the test) were 2.32 times (95% CI 1.46, 3.22) more likely to have been involved in a self-reported crash within the previous five years than those with faster response times. This likelihood ratio became 2.37 (95% CI 1.49, 3.28) when driving exposure was controlled for. As a comparison, individuals who failed a test of useful field of view were 2.70 (95% CI 1.44, 4.44) times more likely to crash than those who passed. The hazard perception test and the useful field of view measure accounted for separate variance in crash involvement. These findings indicate that hazard perception testing and training could be potentially useful for road safety interventions for this age group.

Predicting vitamin D deficiency in older Australian adults

OBJECTIVE There has been a dramatic increase in vitamin D testing in Australia in recent years, prompting calls for targeted testing. We sought to develop a model to identify people most at risk of vitamin D deficiency. DESIGN AND PARTICIPANTS This is a cross-sectional study of 644 60- to 84-year-old participants, 95% of whom were Caucasian, who took part in a pilot randomized controlled trial of vitamin D supplementation. MEASUREMENTS Baseline 25(OH)D was measured using the Diasorin Liaison platform. Vitamin D insufficiency and deficiency were defined using 50 and 25 nmol/l as cut-points, respectively. A questionnaire was used to obtain information on demographic characteristics and lifestyle factors. We used multivariate logistic regression to predict low vitamin D and calculated the net benefit of using the model compared with 'test-all' and 'test-none' strategies. RESULTS The mean serum 25(OH)D was 42 (SD 14) nmol/1. Seventy-five per cent of participants were vitamin D insufficient and 10% deficient. Serum 25(OH)D was positively correlated with time outdoors, physical activity, vitamin D intake and ambient UVR, and inversely correlated with age, BMI and poor self-reported health status. These predictors explained approximately 21% of the variance in serum 25(OH)D. The area under the ROC curve predicting vitamin D deficiency was 0·82. Net benefit for the prediction model was higher than that for the 'test-all' strategy at all probability thresholds and higher than the 'test-none' strategy for probabilities up to 60%. CONCLUSION Our model could predict vitamin D deficiency with reasonable accuracy, but it needs to be validated in other populations before being implemented.

Global analysis of serum microRNAs as potential biomarkers for lung adenocarcinoma

Early diagnosis and the ability to predict the most relevant treatment option for individuals is essential to improve clinical outcomes for non-small cell lung cancer (NSCLC) patients. Adenocarcinoma (ADC), a subtype of NSCLC, is the single biggest cancer killer and therefore an urgent need to identify minimally invasive biomarkers to enable early diagnosis. Recent studies, by ourselves and others, indicate that circulating miRNA s have potential as biomarkers. Here we applied global profiling approaches in serum from patients with ADC of the lung to explore if miRNA s have potential as diagnostic biomarkers. This study involved RNA isolation from 80 sera specimens including those from ADC patients (equal numbers of stages 1, 2, 3, and 4) and age- and gender-matched controls (n = 40 each). Six hundred and sixty-seven miRNA s were co-analyzed in these specimens using TaqMan low density arrays and qPCR validation using individual miRNA s. Overall, approximately 390 and 370 miRNA s were detected in ADC and control sera, respectively. A group of 6 miRNA s, miR-30c-1* (AU C = 0.74; P < 0.002), miR-616(AU C = 0.71; P = 0.001), miR-146b-3p (AU C = 0.82; P < 0.0001), miR-566 (AU C = 0.80; P < 0.0001), miR-550 (AU C = 0.72; P = 0.0006), and miR-939 (AU C = 0.82; P < 0.0001) was found to be present at substantially higher levels in ADC compared with control sera. Conversely, miR-339-5p and miR-656 were detected at substantially lower levels in ADC sera (co-analysis resulting in AU C = 0.6; P = 0.02). Differences in miRNA profile identified support circulating miRNA s having potential as diagnostic biomarkers for ADC. More extensive studies of ADC and control serum specimens are warranted to independently validate the potential clinical relevance of these miRNA s as minimally invasive biomarkers for ADC.

Long non coding RNAs (lncRNAs) are dysregulated in Malignant Pleural Mesothelioma (MPM)

Malignant Pleural Mesothelioma (MPM) is an aggressive cancer that is often diagnosed at an advanced stage and is characterized by a long latency period (20-40 years between initial exposure and diagnosis) and prior exposure to asbestos. Currently accurate diagnosis of MPM is difficult due to the lack of sensitive biomarkers and despite minor improvements in treatment, median survival rates do not exceed 12 months. Accumulating evidence suggests that aberrant expression of long non-coding RNAs (lncRNAs) play an important functional role in cancer biology. LncRNAs are a class of recently discovered non-protein coding RNAs >200 nucleotides in length with a role in regulating transcription. Here we used NCode long noncoding microarrays to identify differentially expressed lncRNAs potentially involved in MPM pathogenesis. High priority candidate lncRNAs were selected on the basis of statistical (P<0.05) and biological significance (>3-fold difference). Expression levels of 9 candidate lncRNAs were technically validated using RT-qPCR, and biologically validated in three independent test sets: (1) 57 archived MPM tissues obtained from extrapleural pneumonectomy patients, (2) 15 cryopreserved MPM and 3 benign pleura, and (3) an extended panel of 10 MPM cell lines. RT-qPCR analysis demonstrated consistent up-regulation of these lncRNAs in independent datasets. ROC curve analysis showed that two candidates were able to separate benign pleura and MPM with high sensitivity and specificity, and were associated with nodal metastases and survival following induction chemotherapy. These results suggest that lncRNAs have potential to serve as biomarkers in MPM.

Identification and Validity of Accelerometer Cut-Points for Toddlers

The purpose of this study was to derive ActiGraph cut-points for sedentary (SED), light-intensity physical activity (LPA), and moderate-to-vigorous physical activity (MVPA) in toddlers and evaluate their validity in an independent sample. The predictive validity of established preschool cut-points were also evaluated and compared. Twenty-two toddlers (mean age = 2.1 years ± 0.4 years) wore an ActiGraph accelerometer during a videotaped 20-min play period. Videos were subsequently coded for physical activity (PA) intensity using the modified Children's Activity Rating Scale (CARS). Receiver operating characteristic (ROC) curve analyses were conducted to determine cut-points. Predictive validity was assessed in an independent sample of 18 toddlers (mean age = 2.3 ± 0.4 years). From the ROC curve analyses, the 15-s count ranges corresponding to SED, LPA, and MVPA were 0–48, 49–418, and >418 counts/15 s, respectively. Classification accuracy was fair for the SED threshold (ROC-AUC = 0.74, 95% confidence interval = 0.71–0.76) and excellent for MVPA threshold (ROC-AUC = 0.90, 95% confidence interval = 0.88–0.92). In the cross-validation sample, the toddler cut-point and established preschool cut-points significantly overestimated time spent in SED and underestimated time in spent in LPA. For MVPA, mean differences between observed and predicted values for the toddler and Pate cut-points were not significantly different from zero. In summary, the ActiGraph accelerometer can provide useful group-level estimates of MVPA in toddlers. The results support the use of the Pate cut-point of 420 counts/15 s for MVPA.

Predictive validity of three ActiGraph energy expenditure equations for children

Purpose This Study evaluated the predictive validity of three previously published ActiGraph energy expenditure (EE) prediction equations developed for children and adolescents. Methods A total of 45 healthy children and adolescents (mean age: 13.7 +/- 2.6 yr) completed four 5-min activity trials (normal walking. brisk walking, easy running, and fast running) in ail indoor exercise facility. During each trial, participants were all ActiGraph accelerometer oil the right hip. EE was monitored breath by breath using the Cosmed K4b(2) portable indirect calorimetry system. Differences and associations between measured and predicted EE were assessed using dependent t-tests and Pearson correlations, respectively. Classification accuracy was assessed using percent agreement, sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve. Results None of the equations accurately predicted mean energy expenditure during each of the four activity trials. Each equation, however, accurately predicted mean EE in at least one activity trial. The Puyau equation accurately predicted EE during slow walking. The Trost equation accurately predicted EE during slow running. The Freedson equation accurately predicted EE during fast running. None of the three equations accurately predicted EE during brisk walking. The equations exhibited fair to excellent classification accuracy with respect to activity intensity. with the Trost equation exhibiting the highest classification accuracy and the Puyau equation exhibiting the lowest. Conclusions These data suggest that the three accelerometer prediction equations do not accurately predict EE on a minute-by-minute basis in children and adolescents during overground walking and running. The equations maybe, however, for estimating participation in moderate and vigorous activity.

Calibration and evaluation of an objective measure of physical activity in pre-school children

Background The purposes of this study were 1) to establish accelerometer count cutoffs to categorize activity intensity of 3 to 5-y old-children and 2) to evaluate the accelerometer as a measure of children’s physical activity in preschool settings. Methods While wearing an ActiGraph accelerometer, 16 preschool children performed five, 3-min structured activities. Receiver Operating Characteristic (ROC) curve analyses identified count cutoffs for four physical activity intensities. In 9 preschools, 281 children wore an ActiGraph during observations performed by three trained observers (interobserver reli-ability = 0.91 to 0.98). Results Separate count cutoffs for 3, 4, and 5-y olds were established. Sensitivity and specificity for the count cutoffs ranged from 86.7% to 100.0% and 66.7% to 100.0%, respectively. ActiGraph counts/15 s were different among all activities (P < 0.05) except the two sitting activities. Correlations between observed and ActiGraph intensity categorizations at the preschools ranged from 0.46 to 0.70 (P < 0.001). Conclusions The ActiGraph count cutoffs established and validated in this study can be used to objectively categorize the time that preschool-age children spend in different physical activity intensity levels.